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Abstract 

A synfire chain is a simple neural network model which can propagate stable synchronous spikes 
called a pulse packet and widely researched. However how synfire chains coexist in one net- 
work remains to be elucidated. We have studied the activity of a layered associative network of 
Leaky Integrate-and-Fire neurons in which connection we embed memory patterns by the Heb- 
bian Learning. We analyzed their activity by the Fokker-Planck method. In our previous report, 
when a half of neurons belongs to each memory pattern (memory pattern rate F = 0.5), the tem- 
poral profiles of the network activity is split into temporally clustered groups called sublattices 
under certain input conditions. In this study, we show that when the network is sparsely con- 
nected {F < 0.5), synchronous firings of the memory pattern are promoted. On the contrary, the 
densely connected network {F > 0.5) inhibit synchronous firings. The sparseness and denseness 
also effect the basin of attraction and the storage capacity of the embedded memory patterns. We 
show that the sparsely(densely) connected networks enlarge(shrink) the basion of attraction and 
increase (decrease) the storage capacity. 

PACS numbers: Valid PACS appear here 
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I. INTRODUCTION 



What is the role of synchronous spikes? It is an important and challenging question 
in neuroscience. In order to approach this question, the ability to generate synchronous 
spikes in various networks have been studied. A synfire chain j]| is a functional feed-forward 
network and able to transmit synchronous spikes called a pulse packet .^Some experimental 
results imply the existence of synfire chains in vivo'M,]^ and in vitro 
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13, 
16| 



5 [. Synfire chains 
15| and have 
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have been also intensively studied theoretically, 0, Q, [s, ^, 
been confirmed to exist in vitro in an iteratively constructed network 

Although Abeles has already mentioned the idea of embedding multiple synfire chains 
in a local network, many of the studies on synfire chains use a homogeneously connected 
network , [lOj. One way to embed multiple synfire chains is to sum up the synaptic 
connections of synfire chains like an associative memory network. Even if each synfire chain 
consists of homogeneous connections, the connections summed them up are inhomogeneous. 
The ability to generate synchronous spikes in homogeneous network has been enthusiastically 
studied, but that in inhomogeneous network have not been throughly investigated. Here we 
embed the chains in the way of associative memory. Therefore we pay attention to whether 
the network transmits synchronous firings of the embedded memory patterns. 

We have previously reported the activity of a network in which the half of neurons join 
in each memory pattern [l^]. In this paper we analyze sparsely and densely connected 



networks and discuss the difference of ability to generate synchronous spikes among those 

nemory constructed by the 
However, the dynamics of 



networks. The sparsely connected networks of the associative memory constructed by the 
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binary neurons have been intensively researched 
layered associative networks with spiking neurons are not throughly studied. 

Section 2 explains the details of our layered associative network, and §3 explains the 
Fokker-Planck method. Section 4 describes the result of our analysis. In §IV At we show 
the activity in single pattern activation. In §IVBI we address the activity in two patterns 
activation. More specifically, we studied the network activity in response to the activation 
of two patterns with different strength but with the same timing ( §IVB ip . and with the 
same strength but with different timing ( §IVB2|) . The basin of attraction is also studied 
( §IVB 3|) . Section [IV CI is devoted for the memory capacity of the network. Section 5 is a 
summary and discussion. 
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FIG. 1: Schematic diagram of our layered associative network. 
II. LAYERED ASSOCIATIVE NETWORK 



In this paper, we consider a layered associative memory network with the standard Heb- 
bian connections 
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23|, l2J]. We constructed feedforward associative networks by using 
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251]. Here a synaptic 



the conventional method of sparsely connected network 
connection J^j from the ith neuron on layer / to the jth neuron on layer / + 1 is given by 



Fil - F)N 



(1) 



where the index of a neuron in a layer is i, j = 1, . . . , A^, and the index of the a memory 
pattern is /i = 1, . . . ,p, and represents the /ith embedded memory pattern of the ith. 
neuron on layer / and takes on a value of either +1 or according to the following probability, 



Prob.[^|'^ = +1] = F. 



(2) 



We call the probability of = +1 the 'pattern rate' F. Then eq. ((Tj) satisfies that the 
expected value of sum of synaptic connection J2i j Aj equals 0, which means that excitation 
and inhibition are balanced regardless of pattern rate F. 

A memory pattern ^^^^ = +1 means that the neuron should fire in memory pattern n 
and = means the neuron should be silent. We analyze the network activity when we 
change the pattern rate F. Figure [1] is a schematic diagram of this network. 

We use the leaky integrate-and-fire (LIF) neuron model, and the dynamics of the mem- 
brane potential can be described as a stochastic differential equation, 

dviit) viit)-V,,,, , /i'"(t) + /o 



dt 



T 



+ 



c 



+ D'r]{t), 



(3) 
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where r is the membrane time constant, Vrest is the resting potential, Iq is the mean of noisy 
input, rj{t) is white Gaussian noise satisfying < r]{t) >— and < rj{t)rj{t') >— 6{t — t'), 
and D' is the amphtude of the noise. Input current l\'°'{t) is obtained by convoluting the 

presynaptic firing ll{t) with the a function a{t) = aHexp{—at) as follows. /3 is a conversion 
constant. 

/•oo 

l'r{t)=P dt'a{t')ll{t-t'). (4) 

^0 

Input current ll{t) is derived from the sum of synaptic connections in which neurons fire. 
To equalize excitatory synaptic inputs for different pattern rate F, we set ll{t) proportional 
to 1/(1 -F). 

^ N n 

3=1 k=l 

where t' ^ indicates the times that the ith neuron on layer / fires. The ith neuron on layer / 
has spiked n times until time t. 

Membrane potential dynamics follow the spike-and-reset rule; when the membrane po- 
tential vl{t) reaches the threshold T^h, a spike is fired, and after the absolute refractoriness 
tref, the membrane potential is reset to the resetting potential K-eset- By implementing 
the absolute refractoriness, burst firings are inhibited and we can focus on pulse packets 
propagation. 

For the following analysis, we introduce the order parameter function m'''^(t), namely the 
overlap, defined by 

^ N n 

"•''"f*) = Fd - F)N " - ^> 5: - U (6) 

^ ^ i=l k=l 

Here, the overlap means how much the firing pattern matches the /ith memory pattern on 
layer I. If neurons with their memory patterns = +1 fire once, then dtrn}''^{t) — 1. 
By using the overlap, ll{t) can be rewritten as 

This means that the synaptic current to a neuron depends only on the overlap of the pre- 
ceding layer and its memory patterns. Here we can see that ll{t) need to be proportional 
to 1/(1 — F) so that the excitatory inputs does not change with different pattern rate F. 
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Throughout this paper, the parameter values are fixed as follows: K-est = K-esct = mV, 
Kh = 15 mV, tref = 1 ms, r = 10 ms, Iq = 0.075 pA, C = 100 pF, D' = 1, a = 2 ms-\ 
and (3 = 0.17 pA. The number of neurons per layer is set to iV = 5000 in whole the LIF 
simulations. 



III. FOKKER-PLANCK METHOD 



In this section, we introduce the analytical method of calculating the membrane potential 
distribution. First, we define a vector whose elements are memory patterns of the ith neuron 
as = (^|'\ ^1'^, . . . , ^1'^). Each element takes on a value +1 or 0, and thus this vector has 
2P combinations. We can define 2^ groups according to ^- values. We call each group a 
sublattice and we discriminate each sublattice with the vector ^ = l^^,^"^, . . . ,C,^). Each 
element ^'^ takes on +1 or values. Here we define -D(^) as the ratio of the number of 
neurons belonging to the sublattice $, to the whole number of neurons and D{^) is described 
as follows: 

m = flieF+{i-e){i-F)). (s) 

Neurons belonging to the same sublattice receive the same synaptic current, because the 
synaptic current depends on only the overlaps and its memory pattern (eq. ([7])). The 



distribution o 
equation, [lo|. 



' the membrane potential is known to evolve according to the Fokker-Planck 



id 



g-^P^{v,t) = -^J[{v,t) + 5{V - V;eset)^(^)4(t " tref), (9) 

= - ^—^ ^ + -— j p,M. (10) 

P^{v, t) is the distribution of the membrane potentials of the neurons belonging to sublattice 
^, and t'^(t) = 2PJ|(Vth,^) is the fiow of probability across the threshold Vth per second per 
neuron, that is to say firing rate; the number of spikes per second per neuron. P^{v,t) and 
z/^(t) satisfy the normalization condition: 

dvPi{v,t)+D{i) r'^' dTu\{t-T)=D{i). (11) 

J -oo Jo 
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Here we define the overlap vector, m}{t) = (m''^(t), m''^(t), . . . , m''P(t)). From eqs. (jl]) 
and ([7j), we can describe tlie synaptic current /^'"(t) by using ^ and m'(t) as follows: 

POO 

= /? / rft'«(t)4(t-t'), (12) 

Jo 
P — FT 

where / is the vector whose all elements are 1 and size is p. 

From eq. ([6]), we can describe the overlap m'''^(t) by using firing rate t'|(t) as 

""''"^^^ = F(l-F) 5Z - ^) - (1 - i^F). (14) 



Here we can describe the network dynamics only by using macroscopic parameters 
Pl{v,t), z/J(t), m\t), and l\. 

In this paper, we numerically calculate the Fokker-Planck equation with the Chang- 



Cooper Method 
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28|. At the boundary, we stock the flow at V|;h and add the stocked 



flow before tref to the probability at K-eset (eq. ([9])). 

The description with the Fokker-Planck method is consistent with the LIF simulation 
in the limit of the number of neurons belonging to each sublattice ND{$,) oo. Because 
MIN(D(^)) < 2~P, we restrict the total number of memory patterns to p ~ 0(1). 



IV. RESULT 



A. Activation of a single pattern 

We have previously reported that an input similar to a memory pattern cause the propa- 
gation of synchronous spikes of the memory pattern when the pattern rate F = 0.5 in both 
of LIF simulation and the Fokker-Planck method [l^]. Here we answer whether the situation 
is same or not when the pattern rate F ^ 0.5. 

The initial condition is a stationary distribution for no external input. We activate the 
first layer of the network. For the first layer activation, we consider the virtual layer and 
describe the overlap on the virtual layer of the first memory pattern as a Gaussian function 
with standard deviation a and total volume m^. The volumes of the other memory patterns 
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are set to 0. Throughout this paper, the standard deviation a is always 0.5 ms. 

,riO'^(t) = J v^a""""^ V 2a2 ^ ' (15) 
[ /x^l. 

We calculate the input currents to neurons from eqs. @ and ([7]), membrane potential 
dynamics and firings from eq. (I3l), and the overlaps from eq. ([6]). Dashed lines in Fig. [2] 
show the firing rates of neurons whose first memory pattern .^''^ is +1. Seven layers are 
vertically aligned from top to bottom. Figure [2](a) is the case of sparsely connected network 
F = 0.4, Fig. [2](b) is the case of conventional network F = 0.5, and Fig. [2](c) is the case 
of densely connected network F = 0.6. Synchronous spikes propagate in not only the case 
F = 0.5 but also F = 0.4 and 0.6. 

Next, we apply the Fokker-Planck method. We calculate the input current to the neurons 
belonging to each sublattice from eqs. ( fT2l) and ( fT3l) . the membrane potential distributions 
from eq. iQ, the firing rates from eq. ffTOl) . and the overlaps from eq. f|T^ . Since the 
overlaps of the l)th memory pattern are always 0, it is enough to divide the neurons 
into the sublattices only with the first memory pattern p/Tj]- Therefore, we consider two 
membrane distributions, P\,{v,t) and P[^{y,t)^ and two firing rates, and z/i(t) on each 

layer. The subscript index (+) means ^ = (+1) sublattice and (— ) means ^ = (0) sublattice. 
Solid lines in Fig. [2] shows the firing rates of ^ = (+1), v+{t) on the seven layers vertically. 

Figure [2] suggests that regardless of pattern rate F in the layered associative network 
the spikes of the input memory pattern synchronously propagate under a single pattern 
activation. Figure [2] also shows the consistency between the results of the LIF simulation 
and those of the Fokker-Planck method. 



B. Activation of two patterns 

We have previously reported that neurons belonging to different subllatices fire in different 
timing under the condition of activating two patterns jl7|. The spiking timing splits even 
when the input memory pattern is similar to one of the memory pattern; m} ~ landm? ~ 0. 
Here we address whether the spiking timing splits or not when the pattern rate F 7^ 0.5. 

Here we focus on the two memory patterns, and the other memory patterns have no 
overlaps. Therefore we divide neurons into sublattices according to the signs of the two 
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patterns as well as §IV A[ The sublattices ^ = (+1,+1), (+1,0), (0,+l), and (0,0) are 

respectively described as (++), (H — ), ( — h), and ( ). We accordingly denote the firing 

rates of each sublattice as i^^^_,_(t), z/^_(t), i/L_,_(t), and u[ {t). 

Throughout §IVB[ we consider the following overlaps of the virtual layer. 



,0,1 f I7„„„ ~ ^++)^ 



+ /°_(l-i^)exp(^^^^n> (16) 



it - 1+- 



^^-exp ^^-^ , (17) 



2a2 

m°'^(t) =0 /i^l,2. (18) 

Then from eq. fll3p the input to (++) and (H — ) sublattices on the first layer is simply 
described as follows, 



In §IVBI the results of the LIF simulation are not shown but we confirmed that their 
results are consistent with those of the Fokker-Planck method. 



1. Different Strength of Input 

In §IVB ll we consider the situation that the volume of the overlaps of the first and second 
memory pattern on the virtual layer are respectively set to iv} and m? , and the timing of 
input to (++) and (+-) is set to the same; J°+F+/°_(l-F) = m\ (l-F)(/°_^-/'j'_) = 
and t++ = t+_ = to in eqs. (fT9|) and (l20l) . Therefore eqs. (fT9!) and (!20|) are rewritten as 



= exp ( ) . (22) 



'l-F)m^-Fm^ f {t - tof^ 

Here we focus on the case that the input is similar to the first memory pattern; im} ~ 1 
and rn? ~ 0. If m? = 0, the input to (++) and (H — ) sublattices is same, /++(t) = /+_ and 
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then the first memory pattern propagates in the shape of synchronous firing packet as shown 
is §IV AI (Fig. [2]). We calculate in the case of = 0.9, = 0.1 by using the Fokker-PIanck 
method. Figure [3] shows the firing rates t'^(t). Solid lines indicate the firing rates of (++) 
sublattices and dashed ones are for the firing rates of (H — ) sublattices z/',__(t). The 

pattern rate is F = 0.4 (Fig. EKa)), F = 0.5 (Fig. Mh)), and F = 0.6 (Fig. EJ^c)). When 
the pattern rate F is 0.5 (Fig. [3]^b)), the spikes of (++) and (H — ) sublattices propagates in 



different timing, as previously reported 17|]. When the pattern rate F is 0.4 (Fig. [3t^a)), at 
the beginning, the neurons of (++) and (H — ) sublattices fires in different timing but after 
propagation of several layers they become to fire synchronously. On the contrary, when the 
pattern rate F is 0.6, the timing difference between (++) and (H — ) sublattices becomes 
larger as spikes propagate. 

These results imply that in the network of the pattern rate F < 0.5, that is the sparsely 
connected network, synchronous firing between sublattices is promoted and in that of F > 
0.5, that is the densely connected network, synchronous firing is suppressed. 

The cause of these promotion and suppression of synchronous firing can be understood 
from the input currents to each sublattice /^(t). The input currents are described as follows: 

IW- ^ ^1,1 ^2,1^^^ 

= 2Fz/^+(t) - 2(1 - F)ui_{t) 

+ (l-2F)(z/^^(t) + z/i+(t)), (23) 

1 — 9F 

+ ^f/ uUt) - 2FvUt). (24) 

When the pattern rate F = 0.5, that is 1 — 2F = 0, (++) sublattices do not interact with 
(H — ). Therefore the timing difference caused by the difference of input strength does not 
change during propagation [17]. When the pattern rate F is less than 0.5, that is 1 — 2F has 
positive value, there are excitatory connections from (++) and (H — ) sublattices to (H — ) 
and (++) ones on the next layer respectively. The excitatory connections seem to promote 
the synchronous firing like a synfire chain. That is why the timing difference decreases as 
spikes propagate. On the contrary, when the pattern rate F is more than 0.5, that is 1 — 2F 
has negative value, there are inhibitory connections as well. The inhibitory connection seems 



to suppress the synchronous firing and made the timing difference larger as spikes propagate. 



2. Different Timing of Input 

In §IVB II when the strength of input to sublattices is different, it seems that sparsely 
and densely connected network respectively promote and suppress synchronous firing. Here 
we observe the activity when we set a difference not in the strength but in the timing of 
input to sublattices; /°+ = /°_ = /° in eqs. ([I9l) and ([201). Then eqs. ^ and ([20]) are 
rewritten as 



We calculate in the case that the input to (++) is earlier than that of (H — ) by 1ms, that 
is = t++ + l[ms]. 

Figure m shows the firing rates t'^(t). Solid lines indicate the firing rates of (++) sublat- 
tices and dashed ones are for the firing rates of (H — ) sublattices v\_{t) on the seven 
layers. When the pattern rate F is 0.5 (Fig. IDI^b)), the spikes of (++) and (H — ) propagates 
at the same speed. When the pattern rate F is 0.4 (Fig. Hl^a)) the timing difference becomes 
smaller, and when the pattern rate F is 0.6 the timing difference becomes larger as spikes 
propagate. All the results of Figs. Il](a-c) is consistent with those of §IVB II 

3. Basin of Attraction 

In §IVB II and §IVB2| we have shown that the sparsely and densely connected network 
seems to respectively promote and suppress synchronous firing between sublattices. Here 
we focus on not firing timing but the stability of firing of sublattices. The timing of inputs 
to (++) and (H — ) is set to be same, i.e., t++ = t+_ = to in eqs. ( [T9l) and ( l20i) . Then eqs. 
( JlQll and (l20ll are rewritten as 




(25) 



(26) 




(28) 



(27) 
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We observe the firing rates when we change the input strength of (++) subllatices and 
(H — ) one independently from to 1. If the maximum of the firing rate of a sublattice 
on the fifth layer is more than 600 [Hz], we regard the sublattice fires. Figure is the result 
obtained with the Fokker-Planck method. The vertical axis means the input strength of 
(++) 1°^ and the horizontal axis means that of (H — ) /°_. The black, dark gray, light 
gray and white region in Fig. [5] respectively mean no firing, activity in (++) sublattice is 
propagated, activity in (H — ) sublattice is propagated, and the first memory pattern is fully 
associated and activity in both (++) and (H — ) sublattices are propagated. The pattern 
rate is F = 0.4(a), F = 0.5(b), and F = 0.6(c). When the pattern rate F = 0.4 (Fig. ^a)) 
the region of the first memory pattern is larger than that in the case of F = 0.5 (Fig. [S](b)). 
On the contrary, when the pattern rate F = 0.6 (Fig. E^c)) the region of the first memory 
pattern is smaller than that in the case of F = 0.5 (Fig. [5]^b)). 

These results imply that the sparsely and densely connected network not only promotes 
and suppresses synchronous firing but also enlarges and shrinks the basin of attraction of 
the memory pattern respectively. The cause of enlargement and shrinkage seems to be the 
excitatory and inhibitory connections between sublattices as described in §IVB II Under 
the existence of excitatory connections between (++) and (h — ) sublattices, (++) and (h — ) 
sublattices mutually excite each other. On the other hand, under the existence of inhibitory 
connections, (++) and (H — ) sublattices mutually inhibit each other. 



C. Storage Capacity 

In the binary neurons network it has reported that sparse connection increases the storage 
capacity of memory patterns 1^, 1^, [2^. Here we show the results in the case of the LIF 
neurons. The number of neurons per layer is set to 5000, and we change the total number 
of the memory pattern p from 1 to 500. The input is written by eq. ( ITSi) as well as §IV At 
but the total volume of the input = 1. 

Figure [6] shows the maximum value of the overlap of the input memory pattern on the 
20th layer. This figure suggests that the smaller the pattern rate F, the more stable the 
propagating patterns are. It seems that the result is also caused by the excitatory and in- 
hibitory connections because synchronous firing enlarges the maximum value of the overlap. 
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V. SUMMARY AND DISCUSSION 



In this paper, we studied the activity of a layered associative network constructed by the 
LIF neurons with taking into account of the sparseness of the mem ory patterns. The effect 



of sparseness has been mainly studied in recurrent networks 18|, ll9|, |20|]. In the layered 



network with spiking neurons, memory patterns propagate in the shape of synchronized 



pulse packet. The sparseness increases the storage capacity ( §IVC|) . which coincide with the 



result of the recurrent networks. The sparseness also affects the propagation of synchronous 
pulse packets between sublattices in the feedforward network case. In two patterns activation 
sparse(dense) connection promotes (suppresses) the propagation of synchronous pulse packets 
between sublattices ( §IVBp in the feedforward network of spiking neurons. 

The increase of storage capacity imply the superiority of the sparse connection in the 
feedforward associative networks. On the other hands, the role of synchronous firing observed 
in sparsely connected networks remains to be elucidated. Future studies will be to elucidate 
how the neural networks can use such synchronous propagation of pulse packets in the 
information processing. 
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FIG. 2: Firing rates of the neurons whose first memory pattern ^''""^ is 1 on the vertical layer under 
the activation of a single pattern. The pattern rate is F = 0.4(a), F = 0.5(b), and F = 0.6(c). 
The total volume of input is 0.6. The dashed lines are obtained with the LIF simulation and 
the solid ones with the Fokker-Planck method. We define when mP'^{t) takes a peak value, as 
3(7 = 1.5[ms]. 
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FIG. 3: Firing rates of (++) sublattices solid lines) and (H — ) ones dashed lines) 

on the vertical layer when the strength of activation is different. The pattern rate is F = 0.4(a), 
F = 0.5(b), and F = 0.6(c). The total volume of input of the first memory pattern is 0.9 and 
that of the second memory pattern is 0.1. We define when mP'^{t) takes a peak value, as 
3cr = 1.5[ms]. These results is obtained with the Fokker-Planck method. 
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FIG. 4: Firing rates of (++) sublattices {i'\__^_{t); solid lines) and (H — ) ones (t/'^ (t); dashed lines) 

on the vertical layer when the timing of activation is different. The pattern rate is F = 0.4(a), 
F = 0.5(b), and F = 0.6(c). The size of input 7° is 1. We define when -f++(t) takes a 
peak value, as 3a = 1.5[ms] and t+_ = t++ + l[ms]=2.5[ms]. These results is obtained with the 
Fokker-Planck method. 
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FIG. 5: The firing of each sublattices. The pattern rate is F = 0.4(a), F = 0.5(b), and F = 0.6(c). 
The vertical axis means the input strength of (++) and the horizontal axis means that of 
(H — ) If the maximum of the firing rate is more than 600[Hz] on the fifth layer, we regard 

the sublattice as 'firing'. The black, dark gray, light gray and white region in respectively mean 
no firing, activity in (++) sublattice is propagated, activity in (H — ) sublattice is propagated, and 
the first memory pattern is fully associated and activity in both (++) and (H — ) sublattices are 
propagated. 
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FIG. 6: The maximum value of the overlap of the input memory pattern on the 20th layer. The 
horizontal axis means the rate of the total number of memory pattern p to the number of the 
neurons per layer N. We averaged the overlap over 10 trials and shows the standard deviations as 
the error-bars. 
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